Papers with annotation framework
Modeling Naive Psychology of Characters in Simple Commonsense Stories (P18-1)
Copied to clipboard
| Challenge: | Understanding a narrative requires reasoning about the causal links between the events in the story and the mental states of the characters, even when those relationships are not explicitly stated. |
| Approach: | They propose a new annotation framework to explain naive psychology of story characters as fully-specified chains of mental states with respect to motivations and emotional reactions. |
| Outcome: | The proposed framework provides a baseline performance on several new tasks suggesting avenues for future research. |
CAMAL: A Novel Dataset for Multi-label Conversational Argument Move Analysis (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models that combine CNN and LSTM structures with speaker ID graphs improve the F1-score of our baseline models to detect speakers’ intents by a large margin. |
| Approach: | They propose a conversational multi-label corpus of teaching transcripts for Conversational Argument Move AnaLysis (CAMAL) the dataset includes 165 discussion transcripts facilitated by pre-service teachers and students . |
| Outcome: | The proposed model improves the F1-score of the baseline model to detect speakers’ intents by a large margin. |
Multilingual Supervision Improves Semantic Disambiguation of Adpositions (2025.coling-main)
Copied to clipboard
| Challenge: | a corpus-based cross-linguistic investigation into the lexical semantics of adpositions is conducted . a significant amount of ambiguity and flexibility in their meanings are present in a variety of languages . |
| Approach: | They conduct a corpus-based corpus analysis of adpositions using SNACS . they find distributional differences in a language's adequacy and disambiguation performance . |
| Outcome: | The proposed framework is suited for analyzing adpositions across languages . it provides a framework for a wide-coverage corpus annotation of high-level senses . |
HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work on detecting user disengagement requires hand-labeling many dialog samples . Currently, self-reported user ratings are constrained to a static corpus . |
| Approach: | They propose an efficient annotation framework that denoises dialog samples instead of manually labeling them. |
| Outcome: | The proposed framework improves annotation efficiency significantly and detects user disengagement in two dialog corpora. |
Elections go bananas: A First Large-scale Multilingual Study of Pluralia Tantum using LLMs (2026.eacl-long)
Copied to clipboard
| Challenge: | a large amount of annotated sentences for each feature can be used for in-depth analysis. |
| Approach: | They propose an annotation framework for lexicalization of pluralia tantum . they use an LLM to annotate each instance from the reference corpus . |
| Outcome: | The proposed framework provides useful annotators for semantic, syntactic and sense categories with accuracy ranging from 51% to 89% on a hand-annotated testset. |
DeFaktS: A German Dataset for Fine-Grained Disinformation Detection through Social Media Framing (2024.lrec-main)
Copied to clipboard
| Challenge: | Distinctively curated across various news topics, DeFaktS offers an unparalleled insight into disinformation’s diverse characteristics. |
| Approach: | They propose to annotate every structural component and semantic element of a news piece, eliminating the need for external knowledge sources. |
| Outcome: | The proposed dataset contains 105,855 posts with 20,008 meticulously labeled tweets and eliminates the need for external knowledge sources. |
IRAC: A Domain-Specific Annotated Corpus of Implicit Reasoning in Arguments (2022.lrec-1)
Copied to clipboard
| Challenge: | Using crowdsourcing, we show that models trained with domain-specific implicit reasonings outperform domain-general models in both automatic and human evaluations. |
| Approach: | They propose to create a domain-specific corpus of implicit reasonings annotated for a wide range of arguments and use it to generate models. |
| Outcome: | The proposed corpus outperforms domain-general models in automatic and human evaluations. |
STARC: Structured Annotations for Reading Comprehension (2020.acl-main)
Copied to clipboard
| Challenge: | STARC is an annotation framework for reading comprehension with multiple choice questions . existing annotation frameworks have limited support for reading behavior analyses . |
| Approach: | They propose a new annotation framework for assessing reading comprehension with multiple choice questions . they use a span ablation dataset to demonstrate that it can be leveraged for a key new application . |
| Outcome: | The proposed framework can be leveraged for a key new application for SAT-like reading comprehension materials. |
Towards automatically generating Questions under Discussion to link information and discourse structure (2020.coling-main)
Copied to clipboard
| Challenge: | Questions under Discussion (QUD) are emerging as a useful approach to spelling out the connection between information structure of sentences and nature of discourse. |
| Approach: | They propose a framework for QUD annotation based on explicit pragmatic principles . they propose generating all potentially relevant questions for a given sentence . |
| Outcome: | The proposed framework supports more reliable discourse structure annotation based on explicit questions . but the proposed approach is not robust enough for authentic data . |
Efficient Pairwise Annotation of Argument Quality (2020.acl-main)
Copied to clipboard
| Challenge: | Especially crowdsourcing suffers from assessors having different reference frames to base their judgments on and task instructions being nondescript and therefore unhelpful in ensuring consistency. |
| Approach: | They propose an efficient annotation framework for argument quality that uses a stochastic transitivity model and an effective sampling strategy to infer high-quality labels. |
| Outcome: | The proposed model significantly outperforms existing annotation procedures and offers statistical insights into argument quality. |
MS-LaTTE: A Dataset of Where and When To-do Tasks are Completed (2022.lrec-1)
Copied to clipboard
| Challenge: | Tasks are a fundamental unit of work in the daily lives of people, who are increasingly using digital means to keep track of, organize, triage, and act on them. |
| Approach: | They compile and release a large-scale dataset that captures location and time for tasks and a BERT-fine-tuned model that predicts task co-occurrence. |
| Outcome: | The proposed framework captures location and time, and predicts task co-occurrence with a BERT fine-tuned model outperforming baselines. |
Understanding Client Reactions in Online Mental Health Counseling (2023.acl-long)
Copied to clipboard
| Challenge: | Communication success relies heavily on reading participants’ reactions, but little research is on how listeners' reactions shape trajectories and outcomes of conversations. |
| Approach: | They propose to use client reactions to predict counseling outcomes by using an annotation framework that encompasses counselors’ strategies and client reaction behaviors. |
| Outcome: | The proposed framework can predict counselors' strategies and client reaction behaviors against a large-scale text-based counseling dataset. |
Inference Annotation of a Chinese Corpus for Opinion Mining (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing tools for opinion mining can accurately predict the writer's attitude in simple explicit sentences. |
| Approach: | They propose to define inference, classify different types and provide an annotation framework to analyze the annotation results. |
| Outcome: | The proposed framework defines inference type, polarity and topic and analyzes the results. |
Symptom Identification for Interpretable Detection of Multiple Mental Disorders on Social Media (2022.emnlp-main)
Copied to clipboard
| Challenge: | Mental disease detection (MDD) from social media has suffered from poor generalizability and interpretability due to lack of symptom modeling. |
| Approach: | They propose to annotate a social media corpus of symptom classes related to 7 mental disorders using a knowledge graph and a new annotation framework to facilitate further research. |
| Outcome: | The proposed model outperforms strong pure-text baselines and provides convincing MDD explanations with case studies. |
Edit me: A Corpus and a Framework for Understanding Natural Language Image Editing (L18-1)
Copied to clipboard
Ramesh Manuvinakurike, Jacqueline Brixey, Trung Bui, Walter Chang, Doo Soon Kim, Ron Artstein, Kallirroi Georgila
| Challenge: | a corpus of image edit requests is elicited for real world images, and an annotation framework is developed . evaluators evaluate crowd-sourced annotation as a means of efficiently creating a sizable corpus at a reasonable cost. |
| Approach: | They propose a natural language interface for interacting with an image editing program . they propose an annotation framework for understanding natural language requests . |
| Outcome: | The proposed tool interprets image edit requests and maps them to actionable commands. |
Make Every Letter Count: Building Dialect Variation Dictionaries from Monolingual Corpora (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Dialects exhibit a substantial degree of variation due to the lack of a standard orthography . however, the ability of Large Language Models (LLMs) to process dialects remains understudied . |
| Approach: | They propose a framework for creating dialect variation dictionaries from monolingual data . they use a dataset to examine how well LLMs can judge Bavarian terms as dialect translations . |
| Outcome: | The proposed framework can judge dialects as dialect translations, inflected variants or unrelated forms of a given German lemma. |
A Manually Annotated Resource for the Investigation of Nasal Grunts (2020.lrec-1)
Copied to clipboard
| Challenge: | acoustic annotation of nasal grunts is described in the whole CID corpus of the french language . acculturation of non-lexical conversational sounds has been debated for a long time . |
| Approach: | They propose an annotation framework for nasal grunts of the whole French CID corpus . they characterise acoustic cues and visual cue conventions followed for the annotation . |
| Outcome: | The proposed framework is based on the entire French CID corpus. |
StorySparkQA: Expert-Annotated QA Pairs with Real-World Knowledge for Children’s Story-Based Learning (2024.emnlp-main)
Copied to clipboard
Jiaju Chen, Yuxuan Lu, Shao Zhang, Bingsheng Yao, Yuanzhe Dong, Ying Xu, Yunyao Li, Qianwen Wang, Dakuo Wang, Yuling Sun
| Challenge: | Existing story reading systems fail to capture the nuances of how education experts think when conducting interactive story reading activities. |
| Approach: | They propose to use existing question-answering (QA) datasets to capture experts' annotations and thinking process to construct a story-based annotation framework. |
| Outcome: | The proposed framework captures experts’ annotations and thinking process and can be used to generate 5, 868 expert-annotated QA pairs with real-world knowledge. |
Specifying Genericity through Inclusiveness and Abstractness Continuous Scales (2024.lrec-main)
Copied to clipboard
| Challenge: | Using a pilot study, we created a small but crucial annotated dataset of 324 sentences, demonstrating the framework’s effectiveness in capturing nuanced aspects of genericity. |
| Approach: | They propose a framework for fine-grained modeling of noun phrases' genericity in natural language using a small but crucial annotated dataset of 324 sentences. |
| Outcome: | The proposed framework can be used to model genericity of noun phrases in natural language and can be easily compared with existing binary annotations. |
TACO – Twitter Arguments from COnversations (2024.lrec-main)
Copied to clipboard
| Challenge: | Argument mining aims to identify the structural elements of arguments, denoted as information and inference, in online discourses. |
| Approach: | They propose to use Twitter Arguments to identify structural elements of arguments, denoted as information and inference, in a dataset that uses 1,814 tweets and an annotation framework that incorporates definitions from the Cambridge Dictionary to define and identify argument components. |
| Outcome: | The proposed dataset identifies arguments on Twitter and achieves an 85.06% macro F1 score in detecting arguments. |
tasksource: A Large Collection of NLP tasks with a Structured Dataset Preprocessing Framework (2024.lrec-main)
Copied to clipboard
| Challenge: | Several initiatives release harmonized datasets or provide harmonization codes to preprocess datasets into a consistent format. |
| Approach: | They propose an annotation framework that enables concise, readable, and reusable annotations. |
| Outcome: | The proposed framework outperforms all publicly available text encoders on all tasks. |
Towards Event Extraction with Massive Types: LLM-based Collaborative Annotation and Partitioning Extraction (2025.emnlp-main)
Copied to clipboard
| Challenge: | Event Extraction (EE) is a long-standing target, but lacks an efficient and effective annotation framework to construct the corresponding datasets. |
| Approach: | They propose an LLM-based collaborative annotation framework that refines annotations of triggers from distant supervision and carries out argument annotation. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on the largest EE dataset to date . it achieves the F1 scores of 90% and 85.3% on the human-annotated test set . |